Examining the Value and Quality of Health Economic Analyses: Implications of Utilizing the QHES

OBJECTIVES: To examine the increasing use of health economic studies and practical implications of evaluating their quality utilizing the Quality of Health Economic Studies (QHES) instrument. METHODS: We first reviewed secondary references to examine ways in which health economic analyses are used in different health care settings, the manner in which these data are appraised and evaluated, and their relevance and value in decision making. The QHES, a new instrument designed to support fast, accurate initial assessments of study quality, was then introduced and validated. A case study was performed using the QHES to score the quality of 30 cost-effectiveness studies in gastroesophageal reflux disease (GERD) published since 1985. Areas where additional research could guide efforts to identify and enhance the use of higher-quality cost-effectiveness studies were suggested. RESULTS: Results from the published validation study of the QHES demonstrated the validity of this new instrument. The resulting QHES scores in the case study of GERD papers ranged from 43 to 91 with a mean of 63.6 (SD=14.7). Approximately 27% of the studies rated had scores less than 50, and 27% had scores above or equal to 75. All 30 studies made conclusions and recommendations and justified them based on their study results. Most studies used appropriate cost and health outcome measures. Very few studies stated the perspective of their analysis and reasons for its selection. The majority of the studies did not perform incremental analysis. CONCLUSIONS: An examination of the QHES validation study and the case study in GERD suggests that there is a rationale and potential utility to use a quality scoring system for cost-effectiveness studies. The QHES may play an important role in discriminating higher-quality cost-effectiveness information to enhance decision making. The QHES can also serve as a guideline for conducting and reporting future cost-effectiveness studies, as an aid in the editorial process, and for stratification in systematic reviews. Complex decisions regarding resource allocation rarely rely solely on economic considerations but do increasingly use health economic analyses. To the extent that such analyses are used, the QHES may help ensure that higher-quality analyses receive more analytic attention and greater weight in the decision-making process.

ealth economic analyses are increasingly common in the published literature. 1 They are also increasingly important. Decision makers face growing pressure to optimize value as well as quality of care. To identify technologies and therapies that provide the greatest value, payers, managed care organizations, and regulatory bodies are all beginning to use health economic analyses, typically in the framework of evidence-based decision making.
This trend is especially important in the pharmaceutical and biotechnology sectors worldwide. Manufacturers are increasingly required to demonstrate the economic as well as clinical value of their products. Both published and unpublished economic analyses now inform decisions on purchasing, subsidization, and formulary acceptance of new pharmaceuticals. The demand for these economic analyses comes from public and private organizations and is seen both in the United States and abroad. 2,3 With this broader use has come greater concern about the validity, methodological quality, and utility of health economic analyses as well as the potential for bias and misuse. [4][5][6][7] This is a particular concern because the professionals who must rely on these analyses to guide decisions may not be expert at evaluating them. One possible solution is to devise a mechanism to more easily select the highestquality data for such decision makers to use. The objective of this article is to assess the potential of such a mechanism. To do so, 3 issues are reviewed first: the growing importance of health economic analyses in decision making, how they are used in specific health care settings, and the challenges involved in evaluating their quality. Next, we introduce a newly developed tool for evaluating the quality of health economic analyses. Finally, we examine the value of this new tool in a particular case study as well as the limitations to the approach and areas where additional research is needed.

■■ Why Health Economic Analyses Are Becoming More Common and Increasingly Important
One major objective of health economic analyses is relating the clinical attributes and health outcomes of treatment strategies to their net costs. Such analyses help compare the relative value of competing strategies for medical/surgical care, therapeutic drugs, devices, or diagnostic tests. Thus, they have an obvious role in purchasing, pricing, and formulary decision making.
With drug and device manufacturers funding large numbers of such studies, 8 the supply of health economic analyses is growing. On the demand side, the 1997 U.S. Food and Drug Administration (FDA) Modernization Act I implemented Section 114, which regulates the use of information submitted by pharmaceutical and device manufacturers to drug formulary committees in managed care or similar entities. This code change, too, has spawned

A B S T R A C T
OBJECTIVE: To examine the increasing use of health economic studies and practical implications of evaluating their quality utilizing the Quality of Health Economic Studies (QHES) instrument.
METHODS: We first reviewed secondary references to examine ways in which health economic analyses are used in different health care settings, the manner in which these data are appraised and evaluated, and their relevance and value in decision making. The QHES, a new instrument designed to support fast, accurate initial assessments of study quality, was then introduced and validated. A case study was performed using the QHES to score the quality of 30 cost-effectiveness studies in gastroesophageal reflux disease (GERD) published since 1985. Areas where additional research could guide efforts to identify and enhance the use of higher-quality cost-effectiveness studies were suggested. RESULTS: Results from the published validation study of the QHES demonstrated the validity of this new instrument. The resulting QHES scores in the case study of GERD papers ranged from 43 to 91 with a mean of 63.6 (SD=14.7). Approximately 27% of the studies rated had scores less than 50, and 27% had scores above or equal to 75. All 30 studies made conclusions and recommendations and justified them based on their study results. Most studies used appropriate cost and health outcome measures. Very few studies stated the perspective of their analysis and reasons for its selection. The majority of the studies did not perform incremental analysis.
CONCLUSION: An examination of the QHES validation study and the case study in GERD suggests that there is a rationale and potential utility to use a quality scoring system for cost-effectiveness studies. The QHES may play an important role in discriminating higher-quality cost-effectiveness information to enhance decision making. The QHES can also serve as a guideline for conducting and reporting future cost-effectiveness studies, as an aid in the editorial process, and for stratification in systematic reviews. Complex decisions regarding resource allocation rarely rely solely on economic considerations but do increasingly use health economic analyses. To the extent that such analyses are used, the QHES may help ensure that higher-quality analyses receive more analytic attention and greater weight in the decision-making process.
KEYWORDS: Cost-effectiveness analysis, Quality, Checklist, Guideline J Managed Care Pharm. 2003(9)1: [53][54][55][56][57][58][59][60][61] Examining the Value and Quality of Health Economic Analyses: Implications of Utilizing the QHES renewed interest in health economic analyses. Moreover, major managed care organizations in the United States are requesting more formal economic dossiers to be supplied by manufacturers to support their products' applications for formulary or reimbursement programs. Outside the United States, national and provincial policies are placing greater emphasis on economic evaluations as well. Australia, the United Kingdom, Denmark, Finland, Norway, Portugal, Belgium, the Netherlands, and some Canadian provinces use the value-for-money equation explicitly in purchasing and pricing decisions. 9 As mechanisms for assessing value improve and as decision processes emphasize value, this proliferation of economic analyses is likely to continue.

■■ How Health Economic Evaluations Are Used in the Real World
Published data are scarce, but from our literature review and experience, health economic analyses seem to be used primarily in purchasing and formulary decisions, less often in developing clinical guidelines. 10 Their use in clinical decision making remains unclear and not rigorously explored.

Benefits Coverage (Formulary) Decisions
Managed care, the advent of capitation, and managed formularies to control rising drug spending have all prompted renewed United States interest in assessing the value of pharmaceuticals and other technologies. Government efforts have been limited; the Medicare Coverage Advisory Committee evaluates the coverage of technology by Medicare but does not have a formal statement for the use of health economic evaluation. 11 The private sector has pursued more expansive initiatives. The Academy of Managed Care Pharmacy (AMCP) has adopted guidelines for submitting economic dossiers to help health plans and managed care organizations objectively evaluate therapeutic agents. So have at least 14 health plans. (These guidelines were first issued by Regence BlueShield, Seattle, Washington, in an effort to set an industry standard for including economic data in formulary decisions.) A recent evaluation suggests the guidelines have had measurable impact; over the last 3 years, the percentage of submissions containing an economic model increased from 55% to 78%. 12 Outside the United States, economic analyses are widely used by government payers. In Australia, decisions to place drugs in the Pharmaceutical Benefits Scheme (a publicly funded insurance program) are made by the federal health minister on the advice of the Pharmaceutical Benefits Advisory Committee, which has a technical economics subcommittee. 13 In the United Kingdom, the National Institute for Clinical Excellence was established within the National Health System in 1999 to provide guidance related to the use of new and existing technology. 2 In North America, the Canadian Coordinating Office for Health Technology Assessment and other organizations have issued formal criteria for the conduct and reporting of health economic analyses. In 5 of the 11 Canadian provinces, submission of economic evaluations is a requirement for inclu-sion in the provincial formulary, while, in others, it is encouraged. 14 In analyzing these examples, the influence of health economic evaluations was generally less than expected.

Clinical Practice Guidelines
It seems logical that health economic evaluations would inform the development of clinical practice guidelines (CPGs). Since these evaluations address the effectiveness and efficiency of care, 15 it is apparent that they could inform the practice of evidence-based medicine. 16 Several sources, including the Consensus Statement on the Role of Cost-Effectiveness Analysis in Health and Medicine, recommend that cost-effectiveness analyses be used as an aid to decision makers 17 and that economic data be incorporated into guidelines where possible. 18 One example of how this can be done comes from the third U.S. Preventive Services Task Forces, which in the year 2000 initiated a process for systematically reviewing cost-effectiveness analyses in formulating its recommendations about clinical preventive services. 19 The group also suggested that this framework should be used in evaluating health care services more broadly.
Despite these promising recommendations, research suggests that the actual integration of economic data into CPGs has not yet been achieved at a meaningful level. A recent review of the development process and quality of CPGs noted that one deficiency was the omission of economic data. 20,21 Another recent report determined that economic analyses were infrequently incorporated into CPGs even when high quality, compelling economic data existed before the guideline was developed. 10 It appears that more research is needed on 2 issues: how relevant economic evaluations are to practicing clinicians and what mechanisms work for integrating issues of efficiency into clinical decision making.

■■ Is It Possible to Identify High-Quality Economic Analyses to Inform Decision Making?
Despite the growing use of health economic information, the quality of published analyses remains less than optimal. 4,6,22,23 This is especially problematic because many of those who need to use these analyses are not equipped to critically evaluate their quality. The recent European Network on Methodology and Application of Economic Evaluation Techniques (EUROMET) survey, for instance, suggests that European decision makers often find health economic analyses to be a "black box," even though they are considered increasingly important in decision making. 14 Increasing the "usability" of economic analyses involves several steps. A number of guidelines and tools are being developed to improve the science behind such analyses. The underlying assumption is that if higher-quality studies are used, then better decisions will be made. While this assumption remains unproven, a quantitative approach has been adopted in the appraisal of randomized clinical trials in systematic reviews. 24 A parallel approach in health economic analysis seems worth investigating.
The goal of many such efforts is to improve methodological performance by "producers" of health economic analyses. But there are also several instruments intended for critical appraisal by "consumers." Among these instruments, the British Medical Journal checklist, [25][26][27] the Canadian Guidelines, 28 and the Journal of the American Medical Association user' s guide 26,27 are most commonly used. With all, the goal is to enable more effective interpretation and use of such analyses.
Although such tools have substantial value, they also face barriers to both widespread adoption and to achieving their ultimate value. First, the construct validity (e.g., convergent and discriminant validity) of these tools has not been formally tested. Second, all existing instruments are qualitative, most contain subjective and open-ended items, and none provide a score to enable simple comparison among studies; thus, they require a relatively sophisticated user. Finally, the existing checklists and appraisal criteria assume that each criterion is of equal weight. Overall, then, it is not clear that tools and guidelines can accurately identify high-quality health economic analyses, nor that users without specific expertise can use them to derive the information they need.
One potentially promising solution is to give the clinical staff who support the decision-making process a mechanism to more easily select the highest-quality health economic analyses for consideration (to the extent quality can be measured). Toward that end, we have developed and validated a weighted scoring instrument that simplifies assessment of the quality of health economic evaluations. 29

■■ The Quality of Health Economic Studies (QHES) Instrument
The QHES instrument was designed to evaluate all 3 common types of health economic analyses: cost-minimization, cost-effectiveness, and cost-utility. The instrument emphasizes appropriate methods, valid and transparent results, and comprehensive reporting of results in each study (Table 1). Its 16 criteria were selected by a panel of 8 health economics experts with experience conducting these analyses. Their selection was made from criteria included in 19 existing guidelines and checklists for cost-effectiveness evaluations ( Table 2). Each criterion has a weighted point value ( Were the perspective of the analysis (societal, third-party payer, etc.) and reasons for its selection stated? 4 3.
Were variable estimates used in the analysis from the best available source (i.e., randomized control trial -best, expert opinion -worst)? 8

4.
If estimates came from a subgroup analysis, were the groups prespecified at the beginning of the study? 1

5.
Was uncertainty handled by (1) statistical analysis to address random events, (2) sensitivity analysis to cover a range of assumptions? 9 6. Was incremental analysis performed between alternatives for resources and costs? 6 7.
Was the methodology for data abstraction (including the value of health states and other benefits) stated? 5

8.
Did the analytic horizon allow time for all relevant and important outcomes? Were benefits and costs that went beyond 1 year discounted (3% to 5%) and justification given for the discount rate? 7

9.
Was the measurement of costs appropriate and the methodology for the estimation of quantities and unit costs clearly described? 8

10.
Were the primary outcome measure(s) for the economic evaluation clearly stated and did they 6 include the major short-term was justification given for the measures/scales used?

11.
Were the health outcomes measures/scales valid and reliable? If previously tested valid and reliable measures were not available, was justification given for the measures/scales used? 7

12.
Were the economic model (including structure), study methods and analysis, and the components of the numerator and denominator displayed in a clear, transparent manner? 8

13.
Were the choice of economic model, main assumptions, and limitations of the study stated and justified? 7

14.
Did the author(s) explicitly discuss direction and magnitude of potential biases? 6

15.
Were the conclusions/recommendations of the study justified and based on the study results? 8

16.
Was there a statement disclosing the source of funding for the study? 3

TOTAL POINTS 100
The QHES Instrument random-effects general least-squares regression based on a conjoint analysis of survey results from 120 international health economists. 29 The perfect quality score for a study is 100. The quality score can be calculated by adding up all of the points for questions answered "yes." The QHES was subsequently validated in a survey including 60 experts (30 clinicians and 30 health economists) in 6 disease categories. We asked the experts to rate 3 health economic evaluation articles in their disease category, first using a global assessment (judgment) and then using the new instrument.
Assuming the global assessment of experts is the "gold standard," results of Spearman's rho test (coefficient=0.78, P<0.0001) and the Wilcoxon test (P=0.53) indicated that the QHES has good convergent validity. The result of analysis of covariance (ANCOVA, F3, 146=5.97, P=0.001) implied that the instrument has good discriminant validity 29 as well. These results indicated that the QHES has good overall construct validity.

■■ Perceived Value of the QHES
The perceived value of the QHES, as discerned from the rela- tively small sample of experts in the validation study, seems to vary with the user' s professional background. Experts in health economics (the 180 experts used to develop and validate the QHES) perceived, on average, moderate value in the instrument. This was measured by questions about the potential value of a tool that could provide a quantitative quality score for a published report so that relative quality among reports could be appraised in a more reliable fashion. Of the 180 experts, 156 returned the survey (i.e., a response rate of 87%). Among those, 117 rated the value of such a tool as greater or equal to 3 on a scale of 1 to 5 (1 = "not valuable at all" and 5 = "extremely valuable") with a mean of 3.6 (±1.0) ( Table 3). A total of 84 experts indicated that they would use the tool or recommend it to others versus 39 who said "no" (Table 4). Among users who are not generally expert in evaluating health economic analyses, interest was stronger. A symposium was convened to introduce the QHES at AMCP' s 14th Annual Meeting (Salt Lake City, Utah, 2002). When asked whether they would use the QHES, 67 of the 88 participants (76%) who responded to the question answered "yes." Among the 129 symposium participants, 40% were employed by pharmaceutical manufacturers, 26% by pharmacy benefits management companies, 15% by provider groups or managed care organizations, and the rest by other institutions.
There is another reason that tools or guidelines like the QHES inspire mixed reactions: health economics evidence is only one factor among many shaping policy and formulary decisions. Noneconomic factors such as institutional culture, the influence of the decision makers' medical specialty and education, and political considerations may all play a role. 30 If such factors are seen as prominent in the decision process, economic information-and methods to improve its quality-may seem less vital.

■■ Applying the QHES: A Case Study
To better understand the potential application of the QHES, we undertook a small case study, examining 30 cost-effectiveness analyses that compared care strategies in gastroesophageal reflux disease.  (The studies, published after 1985, were identified through a search of PubMed.) Rating the studies with the QHES produced scores ranging from 43 to 91 with a mean of 63.6 (SD=14.7). Approximately 27% of the studies rated had scores less than 50 (n= 8), while another 27% had scores above or equal to 75 (n= 8) ( Table 5). The studies having scores below 50 were conducted outside the United States, mainly before 1996 by researchers without academic affiliations, and did not disclose their source of funding. Those scored at 75 or above were generally conducted in the United States after 1996, and all were performed by researchers with academic affiliations. Table 6 presents information regarding how frequently each QHES criterion was met by the 30 studies. All studies did a reasonable job in drawing and justifying conclusions and recommendations based on the study results. Most of them (97%) chose valid and reliable outcome measures or provided justifications for use of previously unvalidated measures. When conducting subgroup analysis, the groups were usually prespecified (93%). Most studies (90%) measured costs appropriately and clearly described the quantities used and unit costs. Surprisingly, only a few studies (13%) performed incremental analysis. The perspective of the analysis and reasons for its selection were stated in www.amcp.org Vol. 9    only 27% of these studies. The method of data abstraction was stated in 37% of these studies; direction and magnitude of potential biases were explicitly discussed in 40%. More than half of the studies (60%) did not disclose the source of funding.

■■ Possible Applications for the QHES
We believe that the quantitative score available with the QHES may enable a variety of users to better judge the relative quality of different studies and to facilitate the decision-making process. It might, for example, streamline the production of the systematic reviews that have become the standard "evidence-based" approach to topic review (supplanting the previous "narrative" reviews from experts). A research team performing such a review might use QHES scores to quickly and accurately stratify studies by quality level (e.g., scores <75 versus >75), as is frequently done in metaanalyses of randomized clinical trials. Similarly, a journal editor confronted by several economic analyses on similar topics might choose to review only those with scores above 50.
The QHES may be especially beneficial to the clinical staff that supports decision makers on Pharmacy and Therapeutics committees. If the P&T committee was reviewing a therapeutic class, the clinical staff could use this tool, at a minimum, to categorize stud-ies as either low or high quality. Even this "blunt" categorization may increase the efficiency of the evaluation process, allowing firstline evaluators to optimize the number of economic analyses actually used to inform the formulary or coverage decisions; it could also help ensure that higher-quality studies play a larger role in the decision-making process. In each of these potential real-world scenarios, the value of the QHES or similar tool would be enabling the end-user to concentrate efforts on a more thorough evaluation and interpretation of the highest-quality data.

■■ Limitations of the QHES Approach and the Case Study
Clearly, widespread adoption of the QHES would require pilot testing the applicability of the tool in several different settings. In addition to the lessons that remain to be learned from such tests, it is important to acknowledge the recognized limitations of any critical appraisal method or scoring instrument as well as limitations specific to the QHES.
First, while few studies have evaluated the use of checklists compared to scoring systems for economic analyses, this topic has generated considerable debate related to the critical appraisal of randomized clinical trials. The debate is largely focused around the reliability and validity of the checklists to truly Questions Frequency %

1.
Was the study objective presented in a clear, specific, and measurable manner? 23 77

2.
Were the perspective of the analysis (societal, third-party payer, etc.) and reasons for its selection stated? 8 27

3.
Were variable estimates used in the analysis from the best available source (i.e., randomized control trial -best, expert opinion -worst)? 21 70 4. If estimates came from a subgroup analysis, were the groups prespecified at the beginning of the study? 28 93

5.
Was uncertainty handled by (1) statistical analysis to address random events, (2) sensitivity analysis to cover a range of assumptions? 14 47 6.
Was incremental analysis performed between alternatives for resources and costs? 4 13

7.
Was the methodology for data abstraction (including the value of health states and other benefits) stated? 11 37 8. Did the analytic horizon allow time for all relevant and important outcomes? Were benefits and costs that went beyond 1 year discounted (3% to 5%) and justification given for the discount rate? 15 50 9.
Was the measurement of costs appropriate and the methodology for the estimation of quantities and unit costs clearly described? 27 90

10.
Were the primary outcome measure(s) for the economic evaluation clearly stated and did they include the major short-term, long-term, and negative outcomes? 22   measure study quality, the ability to capture elements of study quality as opposed to study reporting, and the utility of a score compared to a more comprehensive checklist. [61][62][63][64] We recognize that simplified checklists or scoring tools cannot replace a detailed review of the study methods by those with requisite economics and clinical expertise. This was evident in our case study in which we rated 30 cost-effectiveness analyses in gastroesophageal reflux disease and reported the results (e.g., their scores and frequency in meeting each criterion). However, one application of the QHES is to facilitate a more detailed review by providing an efficient screening mechanism to identify the highestquality studies so that expert reviewers can concentrate their attention on these. Since another possible use would be to help nonexpert users identify higher-quality studies, it is important to assess the inter-rater reliability among nonexperts and to compare the QHES score to a detailed review among nonexperts.
Second, further research is needed to determine the impact of these tools on the results of clinical and policy decisions. In order for quality assessment to become part of the use of economic evaluations, it must be demonstrated that consumers can use the tools to discriminate high-quality analyses from others and, more importantly, that the "use" of higher-quality economic analyses will result in optimal decisions. Third, there is currently a temporal problem in applying such tools to formulary decisions. These tools require that the health economic evaluation be published, or at least be available in relatively final manuscript form, to permit scoring. In our experience, very few cost-effectiveness analyses for formulary applications have been accompanied by a published paper or a final manuscript. The typical case for new drugs (including new chemicals/biologics and new forms of existing chemicals/biologics) is that there is a detailed description of the economic evaluation within the submitted dossier, accompanied by a spreadsheet model. Two limitations are specific to the QHES. One is that this instrument employs yes/no responses rather than a continuous scale for each criterion. In practice, studies often fail to perfectly meet those criteria, but awarding them zero points on that measure seems unlikely to accurately convey the quality associated with each criterion. The other limitation is that some users might not have the knowledge or experience to determine whether studies are properly characterized on the dimensions evaluated by the QHES. For example, we have seen studies stating that models were constructed from the societal perspective but that did not include the impact of productivity loss in either the costs or effectiveness measures. Some users might erroneously give such studies credit in using the QHES since the perspective was stated clearly, although inaccurately.

■■ Discussion
In a wide range of settings worldwide, economic analyses are viewed as valuable tools for incorporating cost considerations into evidence-based clinical decisions. Tools like the QHES may play an important role in enhancing the value of such analyses.
On the most basic level, cost-effectiveness evaluations and other economic analyses should be methodologically sound, clinically oriented, and policy relevant. With ever more such studies being submitted, journal editors and reviewers need tools to more efficiently and reliably identify high-quality studies. The QHES could enable them to make faster, less-subjective decisions regarding the peer-review process and thus enhance the quality of studies published.
In managed care, the QHES could improve the efficiency of P&T review, the objectivity of the process, and the resulting decisions. Although the tool may be of limited use for decisions about new therapies (since published data may be scarce until several months after the therapy' s introduction to the market), it could play an important role in routine formulary evaluations. For example, most formulary review processes include an annual review of the top 15 to 20 therapeutic classes. In this case, the instrument might be used to score the host of economic data on the impact of established therapies; this could provide important insights for keeping formularies current as research accumulates over time.
A practical weighted scoring instrument such as the QHES may also make the economics literature truly accessible to a wider and more diverse audience, allowing users of the literature at all levels to be more informed "consumers." Finally, we hope that with the advent of such a tool, authors of cost-effectiveness studies will pay more attention to many threats to the internal and external validity of their studies early in the design phase.

■■ Further Research Opportunities
The issues reviewed in this paper, and our experience with the QHES, suggest several steps to advance the field and enhance the use of such tools. First, with growing use of economic studies by a broad audience often including nonspecialists, it is important to increase awareness of both the quality variation in published studies and the potential a weighted instrument has to help consumers identify valid, high-quality economic data to support decision making. This awareness could be created by collaborating with national organizations such as AMCP to emphasize methodological quality and to encourage managed care organizations and payers to use tools that help them identify high-quality evaluations. Web-based or other tools could be developed to facilitate the use of the tool and to collect and share the scores assigned to different papers. If this practice were adopted, it might inspire manufacturers to submit more formal presentations or to draft write-ups of their economic analyses that accompany the dossiers submitted to health plans.
Second, input is needed from a wide range of potential users to enhance the scoring tool, increasing both its overall validity and ease of use. While the QHES was validated using experts, it requires further testing and refinement in the "field": among formulary P&T committees, peer reviewers and editors, and those performing systematic reviews. Only this type of scrutiny will reveal whether the tool is improving the use of information, or even improving decisions based on economic analyses. For example, a case study of the actual decision process used by pharmacy directors in a specific therapeutic area would be very helpful, as would an evaluation of how the decision process varies for pharmaceuticals versus medical devices.

■■ Conclusion
The allocation of limited health care resources will never depend only on economic considerations, and the professional judgment of experts will always be required in reviewing the economic analyses that do shape these decisions. But in an environment where health economic analyses are being produced in greater numbers, by a wide range of sources, and evaluated by an even broader group, it seems vital to devise tools that focus attention on objective, high-quality analyses.

DISCLOSURES
Funding for this research was provided by TAP Pharmaceutical Products, Inc., Lake Forest, Illinois, and was obtained by authors Joshua J. Ofman and Chiun-Fang Chiou, who are consultants to TAP. Ofman served as principal author of the study. Study concept and design and analysis and interpretation of data were contributed by Ofman, Chiou, and author Sally W. Wade, also a consultant to TAP. Drafting of the manuscript was primarily the work of Ofman and Chiou. Ofman, Chiou, Wade, and authors Sean D. Sullivan, Peter J. Neumann, James M. Henning, and Joel W. Hay contributed to the critical revision of the manuscript and provided administrative, technical, and/or material support. Statistical expertise was contributed by Chiou and Wade.